Add ALL and EXACTLY_ONE array filter operators for MongoDB and Postgres - #323

Can we avoid a runtime check? ArrayIdentifierExpression contains DataType that can be extracted statically.

Done. The native-array branch now resolves the element type from the compile-time type info on the field expression (ArrayIdentifierExpression#getElementDataType() / IdentifierExpression#getDataType()) via PostgresDataType.fromDataType, and only falls back to inferring from the filter values when the field carries no type info (UNSPECIFIED). The runtime jsonb_typeof guard is now retained only for the JSONB/nested-array path, where the value is schemaless and can be JSON null or a non-array at runtime. Added unit tests covering both the compile-time precedence (declared long[] wins over Integer values) and the fallback.

sure, let me update,
for nested arrays, we'd still need runtime check

Oh I see we already have ITs. Can we move them to DocStoreQueryV1Test?

sure, let me move

Done. Moved all ALL/ONE integration tests into DocStoreQueryV1Test as a nested ArrayMatchAllOneOperatorTest class and removed them from ArrayFiltersQueryIntegrationTest. They now run against the shared collections: nested JSONB path (props.colors) on both stores, native array columns (tags/flags) on the flat collection using the existing PostgresArrayTypeProvider (typed + untyped variants), and the JSONB array column on the flat collection.

Nit: Would be better to assert on the actual docIds for 100% confidence.

Done. All ALL/EXACTLY_ONE integration tests in ArrayMatchAllOneOperatorTest now assert exact docId sets via collectNormalizedIds (e.g. assertEquals(Set.of("1", "2", "3", "7"), ...)), with fixture comments noting which ids match and why. This also implicitly pins the NULL/empty-array exclusions, and there is now an explicit testAllAndOneExcludeNullAndEmptyArrays for that guarantee.

If this how we're handling other operators too? Can you validate what happens when the LHS type does not conform to the RHS for an existing operator?

We validated ALL/EXACTLY_ONE type mismatch against existing EQ and IN in this suite (string vs number).

• Mongo EQ and IN: 0 rows, no throw.
• Postgres EQ of "10" on numeric quantity: matches (JSON ->> is text, so "10" equals stored 10).
• Postgres IN of ["10"] on the same field: 0 rows (jsonb containment is typed).

ALL/EXACTLY_ONE with numeric RHS against string array props.colors: 0 rows on both Mongo and Postgres — same family as Mongo EQ/IN and Postgres IN, not Postgres EQ's text coercion. No new error path; mismatch is empty result for this JSON/document case.

Also, what happens in this case for top-level non-array fields?

for Postgres flat native column (item as text) scalar type non-array throws we say "operator does not exist: text @> text[]"

Document JSONB (mongo + postgres item) -> “top-level non-array → no match,”

Should we not use JsonIdentifierExpression here?

JsonIdentifierExpression on nested document collections — NestedPostgresColTransformer rejects it. Nested ITs still use IdentifierExpression.of("props.colors"). Flat JSONB does use JsonIdentifierExpression.of("props", "colors"). Honest reply: we used it where the transformer supports it; document nested path still uses dotted identifiers.

What does IN mean here semantically?

IN is element membership, not a second array-IN.

ALL + IN [a,b]: every RHS value is in the stored array (extras on the array OK).
EXACTLY_ONE + IN [a,b]: stored array length is 1, and that element is in {a,b}.
So EXACTLY_ONE ["red","green"] = singleton red or singleton green, not the two-element array ["red","green"].

Added test and also add the same in PR desc as well

So this is the legacy PG flow, with no support for first-class columns. That's why you'll see everything is accessed via -> accessor (for ex: jsonb_typeof(document->'tags')). What we need is the flat collection flow. You can parse queries for flat collections using:

    PostgresQueryParser postgresQueryParser =
        new PostgresQueryParser(TEST_TABLE, PostgresQueryTransformer.transform(query), new FlatPostgresFieldTransformer());

Can you please add test cases for that? I am actually surprised that I am unable to find a test class that tests parsed queries for flat collections. If that is indeed the case, would you mind creating one? Thanks :)

But do keep these tests

Done. Added flat-collection parser tests using PostgresQueryTransformer.transform(query) + FlatPostgresFieldTransformer, and kept the legacy-flow tests as you asked. Flat coverage now includes: native array columns (testAllOperatorWithNativeArrayField / testOneOperatorWithNativeArrayField, BOOLEAN[] variants, "tags" @> ? / array_length("tags", 1) = 1 AND "tags" && ?), flat JSONB columns via JsonIdentifierExpression (single- and multi-level paths), and compile-time field type vs value-based inference (testAllOperatorPrefersCompileTimeFieldTypeOverValueInference / fallback test).

Let us run these tests on flat collections please: Collection flatCollection = getFlatCollection(dataStoreName);.

Collection collection = getCollection(dataStoreName);: This returns the legacy storage mode PG in and therefore it's using jsonb_typeof guard.

Done. Added testAllAndOneOnNonArrayValueDoNotMatchFlat — same ALL/EXACTLY_ONE no-match assertion on the flat collection via JsonIdentifierExpression.of("props", "brand") (flat JSONB scalar, exercises the flat path instead of the legacy jsonb_typeof document guard).

Should we also validate this for flat?

Done. Added testAllIsOrderIndependentFlat — same fixture as array_match_test.json loaded into a temp flat table with a native tags TEXT[] column, run with both WITH_TYPE and WITHOUT_TYPE (PostgresArrayTypeProvider). ALL ["blue","red"] matches ids {1,2} on the flat native-array path too.

Same as above, validate for flat?

Done. Added testAllWithDuplicatesInDocumentArrayFlat (temp flat table, native tags TEXT[], WITH_TYPE/WITHOUT_TYPE). ALL ["red"] matches ids {1,2,3,7} — {red,red} matches ALL ["red"] on the flat path as well (@> is element-wise containment).

Same ^

Done. Added testOneWithDuplicatesInDocumentArrayFlat (temp flat table, native tags TEXT[], WITH_TYPE/WITHOUT_TYPE). EXACTLY_ONE ["red"] matches only id 3 — {red,red} is excluded because array_length = 2, same as the document path.

WHERE jsonb_typeof("props"->'colors') = 'array'
  AND jsonb_array_length("props"->'colors') = 1
  AND "props"->'colors' <@ ?::jsonb

Is this better?

Good point — containment (@> and <@) is total on non-array jsonb: it returns false on scalars and NULL on missing keys, never errors. So we've dropped the CASE guard for ALL entirely (WHERE "props"->'colors' @> ?::jsonb) and for the EXACTLY_ONE <@ conjunct. The one place a guard is still required is jsonb_array_length — it raises cannot get array length of a scalar on non-arrays, and since Postgres doesn't guarantee WHERE conjunct evaluation order, an AND-chained jsonb_typeof check isn't a safe guard; the length check keeps a CASE: (CASE WHEN jsonb_typeof(...) = 'array' THEN jsonb_array_length(...) ELSE 0 END) = 1 AND "props"->'colors' <@ ?::jsonb. Unit + integration tests updated.

We still need case @lrathod ?

only for EXACTLY_ONE length check on JSONB fields

jsonb_array_length() raises cannot get array length of a scalar on non-arrays. And we can't just AND-chain jsonb_typeof(x) = 'array' AND jsonb_array_length(x) = 1 because Postgres doesn't guarantee WHERE conjunct evaluation order — the planner may evaluate the length call first. CASE is the only order-forcing construct → kept, but only around the length check: